Parameter-Efficient Model Adaptation for Vision Transformers

In computer vision, it has achieved great transfer learning performance via adapting large-scale pretrained vision models (e.g., transformers) to downstream tasks. Common approaches for model adaptation either update all parameters or leverage linear probes. this paper, we aim study parameter-efficient strategies transformers on the image classification task. We formulate efficient as a subspace training problem and perform comprehensive benchmarking over different methods. conduct an empirical each method focusing its alongside parameter cost. Furthermore, propose framework, which first selects submodules by measuring local intrinsic dimensions then projects them into further decomposition novel Kronecker Adaptation method. analyze compare our with diverse set of baseline methods (including state-of-the-art language models). Our performs best in terms tradeoff between accuracy efficiency across 20 datasets under few-shot setting 7 full-shot setting.